Do you guys have a public IP address for the IPFS node used for the dapp


#1

I’m building an Index of all the FOAM data, using https://thegraph.com/ . I hit the smart contracts and get all the data, but now I realize I also need to retrieve the IPFS data stored about the signals, listings, etc.

The problem with IPFS is that, it doesn’t distribute the files super well. To get all the FOAM IPFS files from public gateways results in many many timeouts, so many that it doesn’t really work.

Do you guys have a public node IPFS node that I can connect to? The command to do so would be:

ipfs swarm connect /ip4/35.233.189.126/tcp/4001/ipfs/QmTiXNcqSSV9inzKXzUbVJ4mc8qov1yXqTEWjAemdbLmVS

But with the node id and ip address of your node.

Please let me know. Thanks!


#2

You can try https://map-api.foam.space/poi/ipfs/<Qm… ipfs multihash>


#3

Hmm that doesn’t appear to work, maybe there is something wrong in the url?

But regardless for what I am doing, I actually need to connect through ipfs connect swarm, I can’t use the HTTP endpoint of an IPFS node.

Is the foam IFPS node open for swarm connections?


#4

We run several IPFS nodes in a cluster for fault-tolerance, and due to the way our infrastructure is set up there is no guarantee that any of our nodes’ multiaddresses given today for a direct connection will still be valid in the future. Our nodes are all still accessible through the “public” IPFS swarm as can be seen by the fact that any POI created through the map interface can also be pulled via, say, the public IPFS gateway at https://gateway.ipfs.io/


#5

In spirit I support the enthusiasm of @dkaj to document the FOAM Ecosystem. I believe that the road forward to success will have to treat the ecosystem as the emergent distributed platform we imagine it becoming. As such, the nascent configuration of FOAM points now cannot be taken as the distributed platform of the future.

With that in mind, is it possible @dkaj for you to program some intelligence into how you are collecting this data to build the graph?

Thanks


#6

Okay, that’s fair, seems like it would be difficult to accommodate opening up a swarm connection with how its set up right now!

Normally I would use the HTTPS endpoint that you guys provide, but for The Graph software it won’t work right now.

Also it’s unfortunate that IPFS doesn’t work as well as I wish it could. I went through this when Indexing all the Origin Dapp data. I think because IPFS is not incentivized, and is a service people opt into, it doesn’t distribute content very well. It’s really hard to get all the data from either Origin, Foam, or other smart contract platforms, without a direct connection. Public gateways seem to always eventually find the files, but it can take 0-15 minutes, and with 1000’s of IPFS hashes it doesn’t scale. (Hopefully filecoin can fix such a problem!)

@yinzeus There is a way around this. Essentially it would be - hit FOAMs http endpoint, grab the file, hash the file myself, confirm that the file hash matches the hash used in the URL. This roundabout way of doing it is to prevent false documents being provided through the HTTP endpoint. Obviously I highly doubt FOAM would post fake docs, but I need to build it properly from the start. Swarm provides a guarantee that the file served matches the hash.

But I probably won’t end up doing this, as it would involve adding some features that The Graph software doesn’t have right now, and there are other subgraphs to index right now :grin: .

@Ryan_foam let me know if there is any plan to open an IPFS swarm connection in the future. If not, thanks anyways!


#7

@dkaj Thanks in advance for this work! When I first heard of TheGraph last month or so, I thought that FOAM index would be a great addition to the ecosystem and also an extra bit of publicity for both FOAM and TheGraph. Haven’t had time to dive in though.

It’s a pity that FOAM IPFS content is not very well distributed. Looks like this important piece of FOAM dApp is pretty centralized for now.


#8

It is a critique valid to almost every project that uses IPFS. IPFS should really be seen as in interim solution right now, it is more of a stepping stone to Filecoin, or something like it.

Right now most decentralized projects are working on their product, and less worried about how IPFS works. And I think with my work at the graph, I’m probably one of the first people to actually try to get IPFS files in a decentralized way, rather than just reading the data through the Dapp that the project has.

I came across this with Origin, and it took a few months before I was actually able to connect to their IPFS node, since they were busy with other stuff.

The funny thing about IPFS is the software isn’t that robust either, swarm connections constantly drop, within minutes sometimes. Currently working on a solution to use linux timers to keep a steady connection to Origins IPFS node


#9

I recognize that you may be pushing the envelope, perhaps in the near future you can provide some guidance for the FOAM project on improvements. I would hope this could be through a grant or bounty so you can provide the assistance in consideration of your expertise!


#10

@dkaj I have a suspicion that there’s something up with your IPFS node’s configuration that is giving you trouble. I just ran a test off my laptop in a limited bandwidth environment (~512KB/s symmetric) to attempt to download our entire IPFS pinset via https://gateway.ipfs.io/ipfs/ and was able to do so completely in about three and a half minutes.

$ time parallel -j64 -a all-ipfs-pins 'curl -sSl https://gateway.ipfs.io/ipfs/{} > downloaded/{}';
parallel -j64 -a all-ipfs-pins   446.39s user 217.05s system 308% cpu 3:35.24 total
$ cat all-ipfs-pins | wc -l
   11053
$ ls downloaded | wc -l
   11053

Do note that this pinset includes both points and challenges, as well as points that have been removed and challenges that have concluded, as well some occasional test data.

I obviously can’t speak to your configuration, but I imagine something’s preventing bitswap/ipfs-swarm from properly working on your node. Do you have the appropriate ports opened on all firewalls between your node and the internet, and are you sure you’re advertising the right node IP, etc.?