Data, data everywhere. Organizations having “big data” and not knowing what to do with it is a huge trend of the 21st century. Here at Global Advantage, we face large amounts of data on a daily basis. Collecting, organizing and structuring data in our day to day workflow, we were experiencing the challenges of “data overload”. To meet the challenge, we developed an online solution that is easy to use and handles vast amounts of information: our own database or what we call our “Data Engine”.
We would like to share our unique process in handling vast amounts of data and the key to structuring and organizing data to make it readily accessible for users. We will go over the technical side, as well as the non-technical side.
Our software developers aimed to optimize our resources when developing a database structure. Diving into to our development research process, we weighed all the options and decided to go with a powerful web framework that harmonizes the relationship between the Visual side (Front-End) and the Database side (Backend). Opposed to developing separate entities, Django, the chosen framework, allows access of all codebase in one Django project. This improves workflow and time efficiency overall.
Django allows for flexibility in choosing a database; in our case, we chose MongoDB. This is a modern NoSQL database that runs off JSON structure and is extremely easy to work with.
Database structure is important for allowing users to sort through and have their data easily accessible and sortable. Developing a tag system, along with a keyword search system, is important for users to be able to find their information. This is not only developed on the back end, but also made accessible from the front end.
After organizing, optimizing and choosing the right language to build our database, the next phase was to design a front-end that is presentable and easy to use. To do this, a good amount of time was dedicated to user research and knowing their needs. User research is an important step to developing a useful application, especially for presenting “big data”.
When we had collected enough information about our users, we then went through the process of designing, prototyping, and light user testing. Through this process, we made multiple changes before reaching the final design. As mentioned before, user inputs collected during the user testing was incredibly valuable. The use of our application depends on our users.
When the final design was decided, we moved onto the development phase. At this stage, our front-end developers started implementing the design to our application and connected it to our database. The development process can take a huge amount of time depending on the application. In the middle of our development, we made sure that went through our beta testing to make sure that our application did not run into any bugs in the final release.
In conclusion, developing an application for storing, organizing, and presenting “big data” can be challenging. However, we were effective because of thorough research and taking the time to know our users. In our company, we made sure to follow these processes to develop the tool we now call “Data Engine.” It is the tool that gives our employees access to the huge amount of data we have collected over the years. We suggest following our blog to learn more about our “Data Engine” and look forward to exciting announcements in the near future!
