TfidfTransformer

Transform a count matrix to a normalized tf or tf-idf representation.

Tf means term-frequency while tf-idf means term-frequency times inverse document-frequency. This is a common term weighting scheme in information retrieval, that has also found good use in document classification.

The goal of using tf-idf instead of the raw frequencies of occurrence of a token in a given document is to scale down the impact of tokens that occur very frequently in a given corpus and that are hence empirically less informative than features that occur in a small fraction of the training corpus.

The formula that is used to compute the tf-idf for a term t of a document d in a document set is tf-idf(t, d) = tf(t, d) * idf(t), and the idf is computed as idf(t) = log [ n / df(t) ] + 1 (if smooth\_idf=False), where n is the total number of documents in the document set and df(t) is the document frequency of t; the document frequency is the number of documents in the document set that contain the term t. The effect of adding “1” to the idf in the equation above is that terms with zero idf, i.e., terms that occur in all documents in a training set, will not be entirely ignored. (Note that the idf formula above differs from the standard textbook notation that defines the idf as idf(t) = log [ n / (df(t) + 1) ]).

If smooth\_idf=True (the default), the constant “1” is added to the numerator and denominator of the idf as if an extra document was seen containing every term in the collection exactly once, which prevents zero divisions: idf(t) = log [ (1 + n) / (1 + df(t)) ] + 1.

Furthermore, the formulas used to compute tf and idf depend on parameter settings that correspond to the SMART notation used in IR as follows:

Tf is “n” (natural) by default, “l” (logarithmic) when sublinear\_tf=True. Idf is “t” when use_idf is given, “n” (none) otherwise. Normalization is “c” (cosine) when norm='l2', “n” (none) when norm=None.

Read more in the User Guide.

Python Reference (opens in a new tab)

Constructors

constructor()

Signature

new TfidfTransformer(opts?: object): TfidfTransformer;

Parameters

Returns

TfidfTransformer

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:35 (opens in a new tab)

Properties

_isDisposed

boolean = false

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:33 (opens in a new tab)

_isInitialized

boolean = false

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:32 (opens in a new tab)

_py

PythonBridge

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:31 (opens in a new tab)

id

string

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:28 (opens in a new tab)

opts

any

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:29 (opens in a new tab)

Accessors

feature_names_in_

Names of features seen during fit. Defined only when X has feature names that are all strings.

Signature

feature_names_in_(): Promise<ArrayLike>;

Returns

Promise<ArrayLike>

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:379 (opens in a new tab)

n_features_in_

Number of features seen during fit.

Signature

n_features_in_(): Promise<number>;

Returns

Promise<number>

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:352 (opens in a new tab)

py

Signature

py(): PythonBridge;

Returns

PythonBridge

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:68 (opens in a new tab)

Signature

py(pythonBridge: PythonBridge): void;

Parameters

Returns

void

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:72 (opens in a new tab)

Methods

dispose()

Disposes of the underlying Python resources.

Once dispose() is called, the instance is no longer usable.

Signature

dispose(): Promise<void>;

Returns

Promise<void>

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:125 (opens in a new tab)

fit()

Learn the idf vector (global term weights).

Signature

fit(opts: object): Promise<any>;

Parameters

Returns

Promise<any>

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:142 (opens in a new tab)

fit_transform()

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit\_params and returns a transformed version of X.

Signature

fit_transform(opts: object): Promise<any[]>;

Parameters

Returns

Promise<any[]>

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:184 (opens in a new tab)

get_feature_names_out()

Get output feature names for transformation.

Signature

get_feature_names_out(opts: object): Promise<any>;

Parameters

Returns

Promise<any>

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:235 (opens in a new tab)

init()

Initializes the underlying Python resources.

This instance is not usable until the Promise returned by init() resolves.

Signature

init(py: PythonBridge): Promise<void>;

Parameters

Returns

Promise<void>

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:81 (opens in a new tab)

set_output()

Set output container.

See Introducing the set_output API for an example on how to use the API.

Signature

set_output(opts: object): Promise<any>;

Parameters

Returns

Promise<any>

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:275 (opens in a new tab)

transform()

Transform a count matrix to a tf or tf-idf representation.

Signature

transform(opts: object): Promise<any[]>;

Parameters

Returns

Promise<any[]>

Defined in: generated/feature_extraction/text/TfidfTransformer.ts:310 (opens in a new tab)