eland.pandas_to_eland#
- eland.pandas_to_eland(pd_df: DataFrame, es_client: Union[str, List[str], Tuple[str, ...], Elasticsearch], es_dest_index: str, es_if_exists: str = 'fail', es_refresh: bool = False, es_dropna: bool = False, es_type_overrides: Optional[Mapping[str, str]] = None, es_verify_mapping_compatibility: bool = True, thread_count: int = 4, chunksize: Optional[int] = None, use_pandas_index_for_es_ids: bool = True) DataFrame #
Append a pandas DataFrame to an Elasticsearch index. Mainly used in testing. Modifies the elasticsearch destination index
Parameters#
- es_client: Elasticsearch client argument(s)
elasticsearch-py parameters or
elasticsearch-py instance
- es_dest_index: str
Name of Elasticsearch index to be appended to
- es_if_exists{‘fail’, ‘replace’, ‘append’}, default ‘fail’
How to behave if the index already exists.
fail: Raise a ValueError.
replace: Delete the index before inserting new values.
append: Insert new values to the existing index. Create if does not exist.
- es_refresh: bool, default ‘False’
Refresh es_dest_index after bulk index
- es_dropna: bool, default ‘False’
True: Remove missing values (see pandas.Series.dropna)
False: Include missing values - may cause bulk to fail
- es_type_overrides: dict, default None
Dict of field_name: es_data_type that overrides default es data types
- es_verify_mapping_compatibility: bool, default ‘True’
True: Verify that the dataframe schema matches the Elasticsearch index schema
False: Do not verify schema
- thread_count: int
number of the threads to use for the bulk requests
- chunksize: int, default None
Number of pandas.DataFrame rows to read before bulk index into Elasticsearch
- use_pandas_index_for_es_ids: bool, default ‘True’
True: pandas.DataFrame.index fields will be used to populate Elasticsearch ‘_id’ fields.
False: Ignore pandas.DataFrame.index when indexing into Elasticsearch
Returns#
- eland.Dataframe
eland.DataFrame referencing data in destination_index
Examples#
>>> pd_df = pd.DataFrame(data={'A': 3.141, ... 'B': 1, ... 'C': 'foo', ... 'D': pd.Timestamp('20190102'), ... 'E': [1.0, 2.0, 3.0], ... 'F': False, ... 'G': [1, 2, 3], ... 'H': 'Long text - to be indexed as es type text'}, ... index=['0', '1', '2']) >>> type(pd_df) <class 'pandas.core.frame.DataFrame'> >>> pd_df A B ... G H 0 3.141 1 ... 1 Long text - to be indexed as es type text 1 3.141 1 ... 2 Long text - to be indexed as es type text 2 3.141 1 ... 3 Long text - to be indexed as es type text [3 rows x 8 columns] >>> pd_df.dtypes A float64 B int64 C object D datetime64[ns] E float64 F bool G int64 H object dtype: object
Convert pandas.DataFrame to eland.DataFrame - this creates an Elasticsearch index called pandas_to_eland. Overwrite existing Elasticsearch index if it exists if_exists=”replace”, and sync index so it is readable on return refresh=True
>>> ed_df = ed.pandas_to_eland(pd_df, ... 'http://localhost:9200', ... 'pandas_to_eland', ... es_if_exists="replace", ... es_refresh=True, ... es_type_overrides={'H':'text'}) # index field 'H' as text not keyword >>> type(ed_df) <class 'eland.dataframe.DataFrame'> >>> ed_df A B ... G H 0 3.141 1 ... 1 Long text - to be indexed as es type text 1 3.141 1 ... 2 Long text - to be indexed as es type text 2 3.141 1 ... 3 Long text - to be indexed as es type text [3 rows x 8 columns] >>> ed_df.dtypes A float64 B int64 C object D datetime64[ns] E float64 F bool G int64 H object dtype: object
See Also#
eland.eland_to_pandas: Create a pandas.Dataframe from eland.DataFrame