Quentin Pradet

@pquentin ·La Réunion ·quentin.pradet.me/blog

205 repos
179 followers
5 following

Python 35%
Shell 18%
TeX 18%
HTML 12%
CSS 6%

2.2k contributions in the last year

12-day longest streak

Jun 2025

222324252627282930

Jul 2025

12345678910111213141516171819202122232425262728293031

Aug 2025

12345678910111213141516171819202122232425262728293031

Sep 2025

123456789101112131415161718192021222324252627282930

Oct 2025

12345678910111213141516171819202122232425262728293031

Nov 2025

123456789101112131415161718192021222324252627282930

Dec 2025

12345678910111213141516171819202122232425262728293031

Jan 2026

12345678910111213141516171819202122232425262728293031

Feb 2026

12345678910111213141516171819202122232425262728

Mar 2026

12345678910111213141516171819202122232425262728293031

Apr 2026

123456789101112131415161718192021222324252627282930

May 2026

12345678910111213141516171819202122232425262728293031

Jun 2026

123456789101112131415161718192021

About Eland is a Python Elasticsearch client for exploring and analyzing data in Elasticsearch with a familiar Pandas-compatible API. Where possible the package uses existing Python APIs and data structures…

About

Eland is a Python Elasticsearch client for exploring and analyzing data in Elasticsearch with a familiar
Pandas-compatible API.

Where possible the package uses existing Python APIs and data structures to make it easy to switch between numpy,
pandas, or scikit-learn to their Elasticsearch powered equivalents. In general, the data resides in Elasticsearch and
not in memory, which allows Eland to access large datasets stored in Elasticsearch.

Eland also provides tools to upload trained machine learning models from common libraries like
scikit-learn, XGBoost, and
LightGBM into Elasticsearch.

Getting Started

Eland can be installed from PyPI with Pip:

bash
$ python -m pip install eland

If using Eland to upload NLP models to Elasticsearch install the PyTorch extras:

bash
$ python -m pip install eland[pytorch]

Eland can also be installed from Conda Forge with Conda:

bash
$ conda install -c conda-forge eland

Compatibility

Supports Python 3.8, 3.9, 3.10 and Pandas 1.5
Supports Elasticsearch clusters that are 7.11+, recommended 8.3 or later for all features to work.

If you are using the NLP with PyTorch feature make sure your Eland minor version matches the minor version of your Elasticsearch cluster. For all other features it is sufficient for the major versions to match.

You need to use PyTorch 1.13.1 or earlier to import an NLP model.

Run pip install torch==1.13.1 to install the aproppriate version of PyTorch.

Prerequisites

Users installing Eland on Debian-based distributions may need to install prerequisite packages for the transitive
dependencies of Eland:

bash
$ sudo apt-get install -y \
  build-essential pkg-config cmake \
  python3-dev libzip-dev libjpeg-dev

Note that other distributions such as CentOS, RedHat, Arch, etc. may require using a different package manager and
specifying different package names.

Docker

Users wishing to use Eland without installing it, in order to just run the available scripts, can build the Docker
container:

bash
$ docker build -t elastic/eland .

The container can now be used interactively:

bash
$ docker run -it --rm --network host elastic/eland

Running installed scripts is also possible without an interactive shell, e.g.:

bash
$ docker run -it --rm --network host \
    elastic/eland \
    eland_import_hub_model \
      --url http://host.docker.internal:9200/ \
      --hub-model-id elastic/distilbert-base-cased-finetuned-conll03-english \
      --task-type ner

Connecting to Elasticsearch

Eland uses the Elasticsearch low level client to connect to Elasticsearch.
This client supports a range of connection options and authentication options.

You can pass either an instance of elasticsearch.Elasticsearch to Eland APIs
or a string containing the host to connect to:

python
import eland as ed

# Connecting to an Elasticsearch instance running on &#39;localhost:9200&#39;
df = ed.DataFrame(&quot;localhost:9200&quot;, es_index_pattern=&quot;flights&quot;)

# Connecting to an Elastic Cloud instance
from elasticsearch import Elasticsearch

es = Elasticsearch(
    cloud_id=&quot;cluster-name:...&quot;,
    http_auth=(&quot;elastic&quot;, &quot;&quot;)
)
df = ed.DataFrame(es, es_index_pattern=&quot;flights&quot;)

DataFrames in Eland

eland.DataFrame wraps an Elasticsearch index in a Pandas-like API
and defers all processing and filtering of data to Elasticsearch
instead of your local machine. This means you can process large
amounts of data within Elasticsearch from a Jupyter Notebook
without overloading your machine.

➤ Eland DataFrame API documentation

➤ Advanced examples in a Jupyter Notebook

python
&gt;&gt;&gt; import eland as ed

&gt;&gt;&gt; # Connect to &#39;flights&#39; index via localhost Elasticsearch node
&gt;&gt;&gt; df = ed.DataFrame(&#39;localhost:9200&#39;, &#39;flights&#39;)

# eland.DataFrame instance has the same API as pandas.DataFrame
# except all data is in Elasticsearch. See .info() memory usage.
&gt;&gt;&gt; df.head()
   AvgTicketPrice  Cancelled  ... dayOfWeek           timestamp
0      841.265642      False  ...         0 2018-01-01 00:00:00
1      882.982662      False  ...         0 2018-01-01 18:27:00
2      190.636904      False  ...         0 2018-01-01 17:11:14
3      181.694216       True  ...         0 2018-01-01 10:33:28
4      730.041778      False  ...         0 2018-01-01 05:13:00

[5 rows x 27 columns]

&gt;&gt;&gt; df.info()

Index: 13059 entries, 0 to 13058
Data columns (total 27 columns):
 #   Column              Non-Null Count  Dtype         
---  ------              --------------  -----         
 0   AvgTicketPrice      13059 non-null  float64       
 1   Cancelled           13059 non-null  bool          
 2   Carrier             13059 non-null  object        
...      
 24  OriginWeather       13059 non-null  object        
 25  dayOfWeek           13059 non-null  int64         
 26  timestamp           13059 non-null  datetime64[ns]
dtypes: bool(2), datetime64[ns](1), float64(5), int64(2), object(17)
memory usage: 80.0 bytes
Elasticsearch storage usage: 5.043 MB

# Filtering of rows using comparisons
&gt;&gt;&gt; df[(df.Carrier==&quot;Kibana Airlines&quot;) &amp; (df.AvgTicketPrice &gt; 900.0) &amp; (df.Cancelled == True)].head()
     AvgTicketPrice  Cancelled  ... dayOfWeek           timestamp
8        960.869736       True  ...         0 2018-01-01 12:09:35
26       975.812632       True  ...         0 2018-01-01 15:38:32
311      946.358410       True  ...         0 2018-01-01 11:51:12
651      975.383864       True  ...         2 2018-01-03 21:13:17
950      907.836523       True  ...         2 2018-01-03 05:14:51

[5 rows x 27 columns]

# Running aggregations across an index
&gt;&gt;&gt; df[[&#39;DistanceKilometers&#39;, &#39;AvgTicketPrice&#39;]].aggregate([&#39;sum&#39;, &#39;min&#39;, &#39;std&#39;])
     DistanceKilometers  AvgTicketPrice
sum        9.261629e+07    8.204365e+06
min        0.000000e+00    1.000205e+02
std        4.578263e+03    2.663867e+02

Machine Learning in Eland

Regression and classification

Eland allows transforming trained regression and classification models from scikit-learn, XGBoost, and LightGBM
libraries to be serialized and used as an inference model in Elasticsearch.

➤ Eland Machine Learning API documentation

python
&gt;&gt;&gt; from xgboost import XGBClassifier
&gt;&gt;&gt; from eland.ml import MLModel

# Train and exercise an XGBoost ML model locally
&gt;&gt;&gt; xgb_model = XGBClassifier(booster=&quot;gbtree&quot;)
&gt;&gt;&gt; xgb_model.fit(training_data[0], training_data[1])

&gt;&gt;&gt; xgb_model.predict(training_data[0])
[0 1 1 0 1 0 0 0 1 0]

# Import the model into Elasticsearch
&gt;&gt;&gt; es_model = MLModel.import_model(
    es_client=&quot;localhost:9200&quot;,
    model_id=&quot;xgb-classifier&quot;,
    model=xgb_model,
    feature_names=[&quot;f0&quot;, &quot;f1&quot;, &quot;f2&quot;, &quot;f3&quot;, &quot;f4&quot;],
)

# Exercise the ML model in Elasticsearch with the training data
&gt;&gt;&gt; es_model.predict(training_data[0])
[0 1 1 0 1 0 0 0 1 0]

NLP with PyTorch

For NLP tasks, Eland allows importing PyTorch trained BERT models into Elasticsearch. Models can be either plain PyTorch
models, or supported transformers models from the
Hugging Face model hub.

bash
$ eland_import_hub_model \
  --url http://localhost:9200/ \
  --hub-model-id elastic/distilbert-base-cased-finetuned-conll03-english \
  --task-type ner \
  --start

The example above will automatically start a model deployment. This is a
good shortcut for initial experimentation, but for anything that needs
good throughput you should omit the --start argument from the Eland
command line and instead start the model using the ML UI in Kibana.
The --start argument will deploy the model with one allocation and one
thread per allocation, which will not offer good performance. When starting
the model deployment using the ML UI in Kibana or the Elasticsearch
API
you will be able to set the threading options to make the best use of your
hardware.

python
&gt;&gt;&gt; import elasticsearch
&gt;&gt;&gt; from pathlib import Path
&gt;&gt;&gt; from eland.common import es_version
&gt;&gt;&gt; from eland.ml.pytorch import PyTorchModel
&gt;&gt;&gt; from eland.ml.pytorch.transformers import TransformerModel

&gt;&gt;&gt; es = elasticsearch.Elasticsearch(&quot;http://elastic:mlqa_admin@localhost:9200&quot;)
&gt;&gt;&gt; es_cluster_version = es_version(es)

# Load a Hugging Face transformers model directly from the model hub
&gt;&gt;&gt; tm = TransformerModel(model_id=&quot;elastic/distilbert-base-cased-finetuned-conll03-english&quot;, task_type=&quot;ner&quot;, es_version=es_cluster_version)
Downloading: 100%|██████████| 257/257 [00:00&lt;00:00, 108kB/s]
Downloading: 100%|██████████| 954/954 [00:00&lt;00:00, 372kB/s]
Downloading: 100%|██████████| 208k/208k [00:00&lt;00:00, 668kB/s] 
Downloading: 100%|██████████| 112/112 [00:00&lt;00:00, 43.9kB/s]
Downloading: 100%|██████████| 249M/249M [00:23&lt;00:00, 11.2MB/s]

# Export the model in a TorchScrpt representation which Elasticsearch uses
&gt;&gt;&gt; tmp_path = &quot;models&quot;
&gt;&gt;&gt; Path(tmp_path).mkdir(parents=True, exist_ok=True)
&gt;&gt;&gt; model_path, config, vocab_path = tm.save(tmp_path)

# Import model into Elasticsearch
&gt;&gt;&gt; ptm = PyTorchModel(es, tm.elasticsearch_model_id())
&gt;&gt;&gt; ptm.import_model(model_path=model_path, config_path=None, vocab_path=vocab_path, config=config)
100%|██████████| 63/63 [00:12&lt;00:00,  5.02it/s]

All public repos (200)

Show forks Show archived