[scikit-learn] ANN Scikit-learn 0.18 released

[Brown J.B.]

Hello community,

Congratulations on the release of 0.19 !
While I'm merely a casual user and wish I could contribute more often, I
thank everyone for their time and efforts!

2016-10-01 1:58 GMT+09:00 Andreas Mueller <t3kcit at gmail.com>:

We've got a lot in the works already for 0.19.
>>
>> * multiple metrics for cross validation (#7388 et al.)
>>
>
I've done something like this in my internal model building and selection
libraries.
My solution has been to have
  -each metric object be able to explain a "distance from optimal"
  -a metric collection object, which can be built by either explicit
instantiation or calculation using data
  -a pareto curve calculation object
  -a ranker for the points on the pareto curve, with the ability to select
the N-best points.

While there are certainly smarter interfaces and implementations, here is
an example of one of my doctests that may help get this PR started.
My apologies that my old docstring argument notation doesn't match the
commonly used standards.

Hope this helps,
J.B. Brown
Kyoto University

 26 class
TrialRanker(object):
 27     """An object for handling the generic mechanism of selecting
optimal
 28     trials from a colletion of trials."""

 43     def SelectBest(self, metricSets,
paretoAlg,
 44
preProcessor=None):
 45         """Select the best [metricSets] by using
the
 46         [paretoAlg] pareto selection object.  Note that it is
actually
 47         the [paretoAlg] that specifies how many optimal [metricSets]
to
 48
select.
 49

 50         Data may be pre-processed into a form necessary for the
[paretoAlg]
 51         by using the [preProcessor] that is a
MetricSetConverter.
 52

 53         Return: an EvaluatedMetricSet if [paretoAlg] selects only
one
 54         metric set, otherwise a list of EvaluatedMetricSet
objects.
 55

 56         >>> from pareto.paretoDecorators import
MinNormSelector
 57         >>> from pareto import
OriginBasePareto
 58         >>> pAlg =
MinNormSelector(OriginBasePareto())
 59

 60         >>> from metrics.TwoClassMetrics import Accuracy,
Sensitivity
 61         >>> from metrics.metricSet import
EvaluatedMetricSet
 62         >>> met1 =
EvaluatedMetricSet.BuildByExplicitValue(
 63         ...           [(Accuracy, 0.7), (Sensitivity,
0.9)])
 64         >>>
met1.SetTitle("Example1")
 65         >>> met1.associatedData = range(5)  # property
set/get
 66         >>> met2 =
EvaluatedMetricSet.BuildByExplicitValue(
 67         ...           [(Accuracy, 0.8), (Sensitivity,
0.6)])
 68         >>>
met2.SetTitle("Example2")
 69         >>> met2.SetAssociatedData("abcdef")  # explicit method
call
 70         >>> met3 =
EvaluatedMetricSet.BuildByExplicitValue(
 71         ...           [(Accuracy, 0.5), (Sensitivity,
0.5)])
 72         >>>
met3.SetTitle("Example3")
 73         >>> met3.associatedData =
float
 74

 75         >>> from metrics.metricSet.converters import
OptDistConverter
 76

 77         >>> ranker = TrialRanker()  # pAlg selects
met1
 78         >>> best =
ranker.SelectBest((met1,met2,met3),
 79         ...                          pAlg,
OptDistConverter())
 80         >>>
best.VerboseDescription(True)
 81         >>>
str(best)
 82         'Example1: 2 metrics; Accuracy=0.700;
Sensitivity=0.900'
 83         >>>
best.associatedData
 84         [0, 1, 2, 3,
4]
 85

 86         >>> pAlg = MinNormSelector(OriginBasePareto(),
nSelect=2)
 87         >>> best =
ranker.SelectBest((met1,met2,met3),
 88         ...                          pAlg,
OptDistConverter())
 89         >>> for metSet in
best:
 90         ...
metSet.VerboseDescription(True)
 91         ...
str(metSet)
 92         ...
str(metSet.associatedData)
 93         'Example1: 2 metrics; Accuracy=0.700;
Sensitivity=0.900'
 94         '[0, 1, 2, 3,
4]'
 95         'Example2: 2 metrics; Accuracy=0.800;
Sensitivity=0.600'
 96
'abcdef'
 97

 98         >>> from metrics.TwoClassMetrics import
PositivePredictiveValue
 99         >>> met4 =
EvaluatedMetricSet.BuildByExplicitValue(
100         ...         [(Accuracy, 0.7), (PositivePredictiveValue,
0.5)])
101         >>> best =
ranker.SelectBest((met1,met2,met3,met4),
102         ...                          pAlg,
OptDistConverter())
103         Traceback (most recent call
last):
104
...
105         ValueError: Metric sets contain differing
Metrics.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-learn/attachments/20161003/66911393/attachment-0001.html>