Responsible Disclosure and Ethical Hacking: An Interview with Dataprovider.com Co-founder Gijs Barends

We provide clear and concise information about what data is collected, how it is collected, and how it will be used. We have sensitive data on asset discovery and security issues, so we have procedures in place to make sure this data doesn’t fall into the wrong hands. We only use data that is publicly available and adheres to robots.txt

We aim to provide a complete picture of the entire web, which means we include every single website we find in our data set, and we use several methods to ensure we capture as much as possible. No one knows how big the web is, so we cannot know how many hostnames we are missing, but by employing several different methods to ensure the broadest coverage, we minimize selection bias. We also have a dedicated data analytics team that continuously evaluates the quality of our data, including the correct definition of data fields and concepts, whether data is representative across countries and languages, the monitoring of any anomalies in the data across time, and comparing machine vs. human classifications. They also ensure any classifiers are trained with representative, balanced data sets.

Having a diverse team also helps. With people from different backgrounds and perspectives, we can better identify potential biases in datasets and take due action to correct them. We conduct regular audits to identify potential biases in datasets, regularly evaluating the data collected to ensure it remains representative over time. This means monitoring changes in web usage patterns and adjusting data collection methods as needed.

We use unbiased data collection methods to avoid inadvertently excluding or underrepresenting certain groups, ensuring that data is collected from diverse sources to accurately reflect the population being studied. This means collecting data from different geographic locations.

Our clients have high expectations for us to operate with a strong sense of ethics and responsibility - they ask this of us contractually. We recognize that promoting ethical and socially responsible outcomes is essential for our long-term success. Therefore, to promote ethical and socially responsible outcomes, we follow a set of guidelines that prioritize fairness, transparency, and accountability. Some of the steps that we take to achieve this include maintaining confidentiality, avoiding harm, transparency, and continual learning We ensure the integrity of our data as we can always trace it to its source: we don’t use any third-party data except for our Traffic Index. Additionally, all our processes, procedures, and policies comply with the European General Data Protection Regulation and the California Consumer Privacy Act.

There are certain companies or countries that are not welcome as a client if we believe they don’t align with our ethical worldview.

Clients can fully rely on Dataprovider.com’s Data Privacy Principles, as almost all data is proprietary, seeing as it is gathered solely by Dataprovider.com themselves. This ensures full compliance with and maintenance of the highest ethical and privacy standards. By conducting ethical impact assessments, we can help identify potential ethical concerns that may arise from using new technologies.

Having an experienced privacy officer is key. In these new cases, there’s no law yet, and often there are no others yet that do the same, so you have to be able to assess the ethical implications yourself.

While we try to capture the entire web, we have programmed our crawler to comply with the international standard of the Robots Exclusion protocol, and we do not crawl websites if there are rules in their robots.txt files that disallow crawling. This data is labeled accordingly in our database.

We also provide anyone with the option to opt-out. We believe innovation and ethical data gathering and provision can go hand-in-hand. They are complementary rather than incompatible.