Sample Header Ad - 728x90

Which Azure Storage / Database Technology to make use of for large scale logging

1 vote
0 answers
66 views
I am looking into making use of Azure hosting services in order to develop a logging service for our company. Azure has a number of storage options available (I have listed links below). However I am not sure which one would be best suited for my needs. I am coming from a MS SQL Server background so I am probably not as open minded as I should be, hence my question here is to try and explore other more suitable options. If any one knows of any place where I can get design help specifically for azure please let me know, as I have not been able to find any. Requirements: Background: We are looking to log messages from a variety of types of devices which the company owns. This needs to be on a mass scale. Several million entries per day. As much as I have tried to keep the design a fixed structure, due to the variety of device types out their they messages differ completely and have different elements. I would somehow like to still be able to query message specific attributes. (How one would do this, I do not know). - All messages come into this logging service as A JSON object. The JSON object obviously varies based on the type of device which sent the message and the type of message the device is sending. - There will be some common attributes such as deviceID, LogTime,customerId etc. - Due to the mention of above, it seems that a document / object store technology would suit the application better. However as mentioned above I have not have much exposure to these types of technologies. Will it be able to efficiently handle the load, and queries etc. - We will need to be able to store months of log values without slowing down daily queries to a halt. - Data analytic's and big queries should be able to be done on the data after some time of running. I'm not sure if one would make use of a different mechanism for this? Possibly a sort of storage layer and then a big data layer. However my storage mechanism needs to keep this future features in mind. Azure offers the following storage and database services: 1. Sql Database 2. DocumentDb 3. Storage: Blobs, Tables,Queues,Files and Disks 4. StorSimple Hybrid cloud Stroage 5. SQL Data Warehouse I am sorry, I know this question is quite general and possibly open ended. I am not looking for anyone to just give the the complete solution in my hand. I am looking for some guidance in the right direction and experience from people who have already gone down this road before.
Asked by Zapnologica (779 rep)
Oct 29, 2015, 07:43 AM