As you’ve probably noticed by now, almost everything we do at Xylos is connected to collecting, processing and understanding the data that’s all around us. Take Connected Factory and self-service BI, for example: these scenarios revolve around gaining insight into data.
Therefore, it makes sense that these applications need a solid data platform (also called a data warehouse). Only with a robust data platform can you aggregate data from several systems and make them available efficiently.
Each scenario has different requirements and goals. Self-service BI is an excellent example: with this system, you’ll have to pay extra attention to data access. The needs are different depending on the situation Fortunately, Xylos can help you. Phew!
ETL: the way from source to storage.
The way data travels needs to be measured from its source to its storage location. This route is usually referred to as ‘ETL’: Extraction, Transformation, Load. Simply put, this includes the following processes:
- Extraction: capturing data from various systems with specific interface methods and formats.
- Transformation: manipulating, transforming and decoding data until it fits the final data storage format.
- Load: loading the date into storage.
This is what an actual example of ETL in IoT looks like:
Imagine you want to capture and store the data sent by Sigfox. This process could happen as follows:
- Extraction: Azure IoT Hub captures the Sigfox messages.
- Transformation: Azure Stream Analytics decodes and parses the Sigfox signal, so that the information can be saved in separate columns within a database.
- Load: the data are stored in Azure SQL or Azure CosmosDB.
Xylos recommends SaaS and PaaS solutions
As you’ve probably noticed, we’ve mentioned a few Azure components. That’s not a coincidence: Xylos actively recommends SaaS and PaaS Azure components for guidance during project implementations. Why? Because...
- The components are available immediately and don’t require significant up-front or hardware investments.
- The necessary components are immediately scalable. If you don’t need capacity, there’s no usage. It’s as simple as that.
- You use a simple setup of test, development and acceptance environments. Number two applies here as well: you only use what you need.
- You benefit from guaranteed uptime and you always use the latest platform versions.
- Your maintenance and operational tasks are minimal.
- Standard compatibility between components is guaranteed.
- You can depend on easy access and integration with cognitive services and Machine Learning techniques.
Some examples of Azure PaaS and SaaS components:
- Azure IoT Hub
- Azure Stream Analytics
- Azure IoT Central
- Azure SQL
- Azure CosmosDB
- Azure Data Lake
Which components are best suited for your project? That’s what we’ll examine during the analysis phase. To facilitate the start of your project, we’ll work with a ‘start template’ developed by Xylos. This ensures we don’t waste time and we get results quickly.
The Cloud Architect and Data Engineer: real-life superheroes when it comes to data
Before the data analyst (or data scientist) can start working on things like data mining, machine learning algorithms, artificial intelligence... the correct data platform needs to be present and fully functional.
But who takes care of writing and setting up the necessary components? That’s what the Xylos Cloud Architect and Xylos Data Engineer are here for – they’ll tackle the job with gusto. Some examples of their tasks:
- Analysing the current situation (such as existing data sources) and desired situation.
- Working closely with the data analysts, data scientists and the IT team. It’s important to define all conditions and requirements – from the user’s as well as a technical point of view.
- Selecting the right components, taking into account a correct balance between requirements and consumption cost.
- Defining and activating the environment via scripting. This is also referred to as IaC: Infrastructure as Code. This is one of the basic DevOps principles and it makes it possible to create uniform testing, development, acceptance and production environments according to a certain version. It also involves simple (de)activation of the environment, so there’s no extra usage cost when a certain activity isn’t necessary.
Do you need a specific data infrastructure or another project? Do you want to temporarily expand your team? Maybe you’d like to add specific skills to your current team through a joint project? It’s all possible. Our Cloud Architects and Data Englineers can’t wait to meet you!
Ready to get started with your new data platform?
Can’t wait to benefit from all these available data hiding in plain sight in your own data platform? Don’t hesitate to contact Xylos. We’re more than happy to help!