asimov-brightdata-module 0.0.4

ASIMOV module for data import powered by the Bright Data web data platform.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153

# ASIMOV Bright Data Module

[![License](https://img.shields.io/badge/license-Public%20Domain-blue.svg)](https://unlicense.org)
[![Package on Crates.io](https://img.shields.io/crates/v/asimov-brightdata-module.svg)](https://crates.io/crates/asimov-brightdata-module)
[![Package on PyPI](https://img.shields.io/pypi/v/asimov-brightdata-module.svg)](https://pypi.org/project/asimov-brightdata-module)
[![Package on RubyGems](https://img.shields.io/gem/v/asimov-brightdata-module.svg)](https://rubygems.org/gems/asimov-brightdata-module)
[![Package on NPM](https://img.shields.io/npm/v/asimov-brightdata-module.svg)](https://npmjs.com/package/asimov-brightdata-module)

[ASIMOV] module for data import powered by the [Bright Data] web data platform.

## ✨ Features

- Imports structured data from Airbnb, Amazon, Crunchbase, eBay, Facebook,
  Google, Indeed, Instagram, LinkedIn, Walmart, X (aka Twitter), Yahoo, and
  YouTube.
- Collects the raw JSON data via the Bright Data API (requires an API key).
- Constructs a semantic knowledge graph based on the [KNOW] ontology.
- Supports plain JSON output as well as [RDF] output in the form of JSON-LD.
- Distributed as a standalone static binary with zero runtime dependencies.

## 🛠️ Prerequisites

- [Rust] 1.85+ (2024 edition) if building from source code

## ⬇️ Installation

### Installation from PyPI

```bash
pip install -U asimov-brightdata-module
```

### Installation from RubyGems

```bash
gem install asimov-brightdata-module
```

### Installation from NPM

```bash
npm install -g asimov-brightdata-module
```

### Installation from Source Code

```bash
cargo install asimov-brightdata-module
```

## 👉 Examples

```bash
export BRIGHTDATA_API_KEY="..."
```

### Fetching X Profiles

```bash
asimov-brightdata-fetcher https://x.com/bright_init   # JSON
asimov-brightdata-importer https://x.com/bright_init  # JSON-LD
```

### Fetching LinkedIn Profiles

```bash
asimov-brightdata-fetcher https://www.linkedin.com/in/orlenchner/
asimov-brightdata-fetcher https://www.linkedin.com/company/bright-data/
```

### Fetching Crunchbase Profiles

```bash
asimov-brightdata-fetcher https://www.crunchbase.com/organization/brightdata
```

### Fetching Amazon Products

```bash
asimov-brightdata-fetcher https://www.amazon.com/Master-Algorithm-Ultimate-Learning-Machine/dp/0465094279
```

## ⚙ Configuration

### Environment Variables

- `BRIGHTDATA_API_KEY`: (required) the [Bright Data API key] to use

## 📚 Reference

### Installed Binaries

- `asimov-brightdata-cataloger`: discovers entities via the Bright Data API
  _(not implemented yet)_
- `asimov-brightdata-fetcher`: collects JSON data from the Bright Data API
- `asimov-brightdata-importer`: collects and transforms JSON into JSON-LD
  _(not implemented yet)_

### Supported Datasets

Dataset | URL Prefix | JSON | RDF
:------ | :--------- | :--: | :--:
Airbnb | `https://www.airbnb.com/rooms/` | ✅ | 🚧
Amazon | `https://www.amazon.com/` | ✅ | 🚧
  | `https://www.amazon.com/sp?seller=` | ✅ | 🚧
Crunchbase | `https://www.crunchbase.com/organization/` | ✅ | 🚧
eBay | `https://www.ebay.com/itm/` | ✅ | 🚧
Facebook | `https://www.facebook.com/events/` | ✅ | 🚧
  | `https://www.facebook.com/groups/` | ✅ | 🚧
  | `https://www.facebook.com/marketplace/item/` | ✅ | 🚧
  | `https://www.facebook.com/share/p/` | ✅ | 🚧
Google | `https://www.google.com/shopping/product/` | ✅ | 🚧
Indeed | `https://www.indeed.com/cmp/` | ✅ | 🚧
Instagram | `https://www.instagram.com/` | ✅ | 🚧
  | `https://www.instagram.com/p/` | ✅ | 🚧
  | `https://www.instagram.com/reel/` | ✅ | 🚧
LinkedIn | `https://www.linkedin.com/company/` | ✅ | 🚧
  | `https://www.linkedin.com/in/` | ✅ | 🚧
  | `https://www.linkedin.com/jobs/` | ✅ | 🚧
  | `https://www.linkedin.com/posts/` | ✅ | 🚧
  | `https://www.linkedin.com/pulse/` | ✅ | 🚧
Walmart | `https://www.walmart.com/global/seller/` | ✅ | 🚧
  | `https://www.walmart.com/ip/` | ✅ | 🚧
X (Twitter) | `https://x.com/` | ✅ | ✅
Yahoo | `https://finance.yahoo.com/quote/` | ✅ | 🚧
YouTube | `https://www.youtube.com/@` | ✅ | 🚧
  | `https://www.youtube.com/watch?v=` | ✅ | 🚧
<img width="100" height="1"/> | <img width="550" height="1"/> | <img width="50" height="1"/> | <img width="50" height="1"/>

## 👨‍💻 Development

```bash
git clone https://github.com/asimov-modules/asimov-brightdata-module.git
```

---

[![Share on X](https://img.shields.io/badge/share%20on-x-03A9F4?logo=x)](https://x.com/intent/post?url=https://github.com/asimov-modules/asimov-brightdata-module&text=asimov-brightdata-module)
[![Share on Reddit](https://img.shields.io/badge/share%20on-reddit-red?logo=reddit)](https://reddit.com/submit?url=https://github.com/asimov-modules/asimov-brightdata-module&title=asimov-brightdata-module)
[![Share on Hacker News](https://img.shields.io/badge/share%20on-hn-orange?logo=ycombinator)](https://news.ycombinator.com/submitlink?u=https://github.com/asimov-modules/asimov-brightdata-module&t=asimov-brightdata-module)
[![Share on Facebook](https://img.shields.io/badge/share%20on-fb-1976D2?logo=facebook)](https://www.facebook.com/sharer/sharer.php?u=https://github.com/asimov-modules/asimov-brightdata-module)
[![Share on LinkedIn](https://img.shields.io/badge/share%20on-linkedin-3949AB?logo=linkedin)](https://www.linkedin.com/sharing/share-offsite/?url=https://github.com/asimov-modules/asimov-brightdata-module)

[ASIMOV]: https://github.com/asimov-platform
[Bright Data]: https://brightdata.com/products/web-scraper
[Bright Data API key]: https://docs.brightdata.com/general/account/api-token
[JSON-LD]: https://json-ld.org
[KNOW]: https://github.com/know-ontology
[NPM]: https:/npmjs.org
[Python]: https://python.org
[RDF]: https://github.com/rust-rdf
[Ruby]: https://ruby-lang.org
[Rust]: https://rust-lang.org