Skip to main content
Jan 18, 2022 9:10 AM UTC
Solved

Which is the most efficient way for retrieve entities? TWX 8.5

Hello!,

 

I'm developing a very huge aplication and we will have a lot of devices (more than 1 million).

The main thing is that the app has to have the posibility of retrieve one o two from the one million of device when It will be necesary. Which is the most efficient way for to do that?

These are the options that I thougth:

1. GetImplementingThingsWithData it's efficient (with a query parameter)?

2. It's better to have all this data in a database and make a query? 

3. Create a network tree entity it's a good idea?

 

Thank you in advance.

 

Luis :D.

PaiChungAnswer
Jan 20, 2022 6:39 PM UTC

Potentially a separate database with optimized views/queries might work better. Difficult to predict.

But else you just make sure you have grouped your Things with your tags so you can 'paginate' through them.

5 replies

Jan 18, 2022 9:22 PM UTC

Options 1 and 3 mean you will load one million rows of data into memory, that isn't a good way to go.

If you can group your devices into smaller numbers you can do an iterative approach or a 'pagination' approach

Database and Query would work

Any which way you do this, I recommend you device some way to 'paginate' so you aren't loading 1 million records at one time.

Jan 19, 2022 8:53 AM UTC

Hello @PaiChung thank you a lot for your answer,

 

One question respect GetImplementingThingsWithData, when you use this fuction, which is the correct option?:

1. First it retieve all entities from a template and filter with the query ( so, first TWX has to load millions of devices)

2. Only retrieve the things that satisfy the query parameter (imagin that I have 1 million devices/things and I want retrieve only the device/thing with id=011, so only load in memory one device).

 

Thank you in advance.

 

Luis

Jan 19, 2022 6:22 PM UTC

If you have 1 million items, I recommend against using GetImplementingThingsWithData you could consider using QueryImplementingThingsWithData

With the GET it will always retrieve all entities and their property values, this would be too much.

With Query I would be careful, there are limited indexes that would allow you to use Query and not retrieve all 1 million directly after which the remainder of the query would be applied. I the case of entities I think only Model Tags would be an indexed field allowing you to retrieve less than your 1 million entities on a first request.

 

All that said, perhaps you can spec your system to have plenty of processing power and memory to handle this, however I recommend designing things such that you are always avoiding pulling 1 million entities and their info.