(port) > Software > YouTube >

https://github.com/philbot9/youtube-comment-scraper/

Downloads the comments on YouTube videos.

Used to be available at ytcomments.klostermann.ca but that IP got banned because of overuse.

WARNING - Do not overuse this program, or maybe you will get IP-banned. The authors' GUI website did.
See my youtube-comment-scraper.sh script for how I use it.

2020-06-09 - 1.0.2 cli on Devuan_beowulf_3.0.0_amd64
2020-05-09 - 1.0.1 cli on Debian 10.1.0-amd64-xfce-CD-1
2019-11-01 - 1.0.1 (cli) on Debian 10.1.0-amd64-xfce-CD-1
2019-11-01 - git (gui) on Debian 10.1.0-amd64-xfce-CD-1
2019-10-13 - git (cli) on Debian 9.9.0-i386-xfce-CD-1
I think I had this working on Windows momentarily, but I changed nothing and it just stopped working after I forgot about it for a while.

Table of Contents [hide]

1 2020-05-28 - 1.0.2 (cli)
2 2019-11-01 - 1.0.1 (cli)
- 2.1 Installation
  - 2.1.1 Running
3 2019-11-01 - git (gui)
- 3.1 Installation (GUI/web browser)
- 3.2 Running
4 2019-10-13 - git (cli)
- 4.1 installation notes
5 Alternatives

2020-05-28 - 1.0.2 (cli) ∞

Horrifically-large comment downloads were failing. See:

Per the author's instructions, I did:

\sudo  \npm  install  -g youtube-comment-scraper-cli@1.0.2
youtube-comment-scraper --format csv --stream -- koPmuEyP3a0 | tee output.csv

✕ API response does not contain a "content_html" field

.. testing.

2019-11-01 - 1.0.1 (cli) ∞

Installation ∞

# Optionally keep a copy..
# \git  clone https://github.com/philbot9/youtube-comment-scraper-cli/
# \cd                                   ./youtube-comment-scraper-cli/

\sudo  \apt-get  -y  install  \
  node.js  \
  npm  \
` # `
# WARNING - as of 2019-11-01 I get:
#   npm WARN npm npm does not support Node.js v10.15.2
\sudo  \npm  install  -g  youtube-comment-scraper-cli

Running ∞

Do note that downloading can take a whole lot of time..

source_video_id='jNQXAC9IVRw'
\youtube-comment-scraper  --format csv  --outputFile "$source_video_id".csv  "$source_video_id"

It seemed too slow, so I switched to the GUI for a while, but the dependencies for the GUI, and running it, is just abysmal!

2019-11-01 - git (gui) ∞

not recommended

Installation (GUI/web browser) ∞

Install various dependencies, see https://github.com/spiralofhope/shell-random/tree/master/zombie/youtube-comment-scraper--install-dependencies.sh

\sudo  \apt-get  -y  install  \
  apt-transport-https  \
  ca-certificates  \
  curl  \
  gnupg2  \
  software-properties-common  \
` # `
\curl  -fsSL https://download.docker.com/linux/debian/gpg  |  \sudo  \apt-key  add -

# optionally verify the docker key is correct:
\sudo  \apt-key  fingerprint 0EBFCD88  |  \grep  '9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88'

\sudo  \add-apt-repository  \
   "deb [arch=amd64] https://download.docker.com/linux/debian/ \
   $(lsb_release -cs) \
   stable"
\sudo  \apt-get  update
\sudo  \apt-get  -y  install docker-ce docker-ce-cli containerd.io

# optionally test docker:
\sudo  \docker  run hello-world

if  [ ! -f '/var/run/docker.pid' ]; then
  \sudo  \dockerd  &
fi
\sudo  \docker  pull mongo

Running ∞

NOTE: You should probably examine youtube-comment-scraper.sh

if  [ ! -f '/var/run/docker.pid' ]; then
  \sudo  \dockerd  2> /dev/null &
fi
\sudo  \docker  run  --name="ytcomments-mongo" --restart="always"  -v  "$PWD":/data/db  -d mongo
\sudo  ./deploy

Visit http://localhost:49161/
- http://0.0.0.0:49161/ also worked for me.

Stopping:

\docker  stop  $( \sudo  \docker  ps -aq )
\docker  rm    $( \sudo  \docker  ps -aq )
# This worked once but stopped working..
\systemctl  stop  containerd
\systemctl  stop  dockerd
# I have no idea how to actually fucking stop docker.
\killall  dockerd  containerd
\rm  /var/run/docker.pid
\umount  /var/lib/docker/overlay2/*

# For some fucked up reason the permissions get messed up..
#\find  /home/user/  -exec \chmod  g+rw       {} \;
#\find  /home/user/  -exec \chown  user:user  {} \;

# See what is running:
\sudo  \docker  ps
\sudo  \docker  restart $container_id
\sudo  \docker  stop    $container_id

# See what is running:
\sudo  \docker  ps
\sudo  \docker  restart $container_id
\sudo  \docker  stop    $container_id

2019-10-13 - git (cli) ∞

some random notes

https://nodejs.org/dist/latest-v0.12.x/

node-ansi-regex
node-marked

not sure:

npm WARN package.json [email protected] Non-dependency in bundleDependencies: imurmurhash

linuxbrew - https://brew.sh/

sh -c "$( curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh )"

.. fails sigh

test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv)
test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)
test -r ~/.bash_profile && echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.bash_profile
echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.profile

to .zshrc

export PATH="$HOME/.linuxbrew/bin:$PATH"
export MANPATH="$HOME/.linuxbrew/share/man:$MANPATH"
export INFOPATH="$HOME/.linuxbrew/share/info:$INFOPATH"

installation notes ∞

(curl, Git, ncurses, Ruby)

#!/usr/bin/env  sh


# FIXME - I still don't have "npm"


# As root:
if ! [ "$USER" = 'root' ]; then
 \echo  "enter root password"
 /bin/su  --command  "$0"
else


#:<<'}'  # NodeJS
         # https://github.com/nodesource/distributions/blob/master/README.md
{
  \apt-get  install  --yes  curl

  # It seems to just be available:
  #\apt-get  install -y nodejs
  # Also possibly:
  #\apt-get  install  --yes  build-essential nodejs
  # The simple manual way was:
  #\curl -sL https://deb.nodesource.com/setup_12.x | bash -
  #\apt-get  install -y  nodejs

  # The more thorough way:

  \curl  --silent  --location  https://dl.yarnpkg.com/debian/pubkey.gpg | \apt-key  add -
  \echo  'deb https://dl.yarnpkg.com/debian/ stable main' | \tee  '/etc/apt/sources.list.d/yarn.list'
  # The following does an apt-get update:
  \curl  --silent  --location  https://deb.nodesource.com/setup_12.x | bash -
  \apt-get  install  --yes  \
    nodejs  \
    yarn  \
  ` # `

  # You may also need development tools to build native addons:
  #\apt-get  install  --yes  gcc  g++  make

  # I don't know why the fuck it doesn't do this already..
  \ln --symbolic  --force  /usr/bin/nodejs  /usr/bin/node


# Other people say these are dependencies, but some are not pulled in from installing nodejs from the repo.
sudo apt-get install build-essential curl git m4 ruby texinfo libbz2-dev libcurl4-openssl-dev libexpat-dev libncurses-dev zlib1g-dev

}


fi   # The above is run as root

(curl, Git, ncurses, Ruby)

#!/usr/bin/env  sh


# FIXME - I still don't have "npm"


# As root:
if ! [ "$USER" = 'root' ]; then
 \echo  "enter root password"
 /bin/su  --command  "$0"
else


#:<<'}'  # NodeJS
         # https://github.com/nodesource/distributions/blob/master/README.md
{
  \apt-get  install  --yes  curl

  # It seems to just be available:
  #\apt-get  install -y nodejs
  # Also possibly:
  #\apt-get  install  --yes  build-essential nodejs
  # The simple manual way was:
  #\curl -sL https://deb.nodesource.com/setup_12.x | bash -
  #\apt-get  install -y  nodejs

  # The more thorough way:

  \curl  --silent  --location  https://dl.yarnpkg.com/debian/pubkey.gpg | \apt-key  add -
  \echo  'deb https://dl.yarnpkg.com/debian/ stable main' | \tee  '/etc/apt/sources.list.d/yarn.list'
  # The following does an apt-get update:
  \curl  --silent  --location  https://deb.nodesource.com/setup_12.x | bash -
  \apt-get  install  --yes  \
    nodejs  \
    yarn  \
  ` # `

  # You may also need development tools to build native addons:
  #\apt-get  install  --yes  gcc  g++  make

  # I don't know why the fuck it doesn't do this already..
  \ln --symbolic  --force  /usr/bin/nodejs  /usr/bin/node


# Other people say these are dependencies, but some are not pulled in from installing nodejs from the repo.
sudo apt-get install build-essential curl git m4 ruby texinfo libbz2-dev libcurl4-openssl-dev libexpat-dev libncurses-dev zlib1g-dev

}


fi   # The above is run as root

another install script I had..

\sudo  \tar  -xf node-v0.12.18.tar.xz   --directory /usr/local --strip-components 1

Alternatives ∞

Last updated 2021-02-20 at 08:59:47

6 comments on “youtube-comment-scraper”

spiralofhope on 2019-10-29 at 9:24 pm said:

Well shit, I guess I need to have a separate 64bit install to get this working. I guess this is the push I needed to transition a chunk of things back to 64bit and leave things stagnant on 32bit just for some old software testing.

spiralofhope on 2019-11-01 at 4:28 pm said:

got both the gui and cli working

spiralofhope on 2019-11-01 at 6:01 pm said:

oh god is the gui fucking with my system ownership and permissions..

the cli works fine (though no progress bar) so I'll use that

Jim on 2020-05-09 at 7:12 am said:

Is there anyway to use this to capture YouTube live chats and filter down to specific users?

- spiralofhope on 2020-05-09 at 7:35 am said:
  
  No.
  
  There is no functionality for downloading the chat section, just the comments.
  
spiralofhope on 2020-06-09 at 3:48 pm said:

updated to 1.0.2

spiralofhope

youtube-comment-scraper

2020-05-28 - 1.0.2 (cli) ∞

2019-11-01 - 1.0.1 (cli) ∞

Installation ∞

Running ∞

2019-11-01 - git (gui) ∞

Installation (GUI/web browser) ∞

Running ∞

2019-10-13 - git (cli) ∞

installation notes ∞

Alternatives ∞

6 comments on “youtube-comment-scraper”

Leave a Reply Cancel reply

Search