github integration

[danhammer]

Currently, on our local clone of this project, we change this line to point to a file called forma_bootstrap_robin.sh, which contains Robin's github credentials. We need this in order to clone the forma-clj project on the cluster. He doesn't want these credentials to go in this repo, which makes sense.

Ideally, we would have a way to configure github using our own credentials which could be stored or copied into the forma-deploy repo on our local machine. Or maybe in ~/.ssh or something. I don't know. We can talk more about this if I am being unclear.

[sritchie]

I think we should never clone the repo onto every machine in the cluster.
See my rsync comment.

On Wed, Jul 18, 2012 at 3:13 PM, Dan <
reply@reply.github.com

wrote:

Currently, on our local clone of this project, we change this line
to point to a file called forma_bootstrap_robin, which contains Robin's
github credentials. We need this in order to clone the forma-clj project
on the cluster. He doesn't want these credentials to go in this repo,
which makes sense.

Ideally, we would have a way to configure github using our own
credentials which could be stored or copied into the forma-deploy repo
on our local machine. Or maybe in ~/.ssh or something. I don't know.
We can talk more about this if I am being unclear.

---

Reply to this email directly or view it on GitHub:
#2

Sam Ritchie, Twitter Inc
703.662.1337
@sritchie09

(Too brief? Here's why! http://emailcharter.org)

[robinkraft]

Note that our bootstrap script has placeholders for your private key and public key git credentials.

I'm not sure what the right way to do this would be.

[danhammer]

Fair enough. Where is the rsync comment, @sritchie?

[sritchie]

Search email for "rsync"

[danhammer]

nothing comes up. paste in the comment here, if you think it's worth the time.

[robinkraft]

Here 'tis

You have to click to expand the comment.

It's still extremely convenient to have the repo on the cluster, for quickly switching between branches, etc. I'm not convinced rsync is the way to go.

[robinkraft]

I don't care about having it on every machine, but having it on the master node is pretty sweet.

[sritchie]

Try rsync once, then judge for yourself. I'm totally convinced.

On Wed, Jul 18, 2012 at 3:59 PM, Robin Kraft <
reply@reply.github.com

wrote:

Here 'tis

You have to click to expand the comment.

It's still extremely convenient to have the repo on the cluster, for
quickly switching between branches, etc. I'm not convinced rsync is the way
to go.

---

Reply to this email directly or view it on GitHub:
#2 (comment)

Sam Ritchie, Twitter Inc
703.662.1337
@sritchie09

(Too brief? Here's why! http://emailcharter.org)

[robinkraft]

Deal!! Great idea.

On Jul 18, 2012, at 4:02 PM, Sam Ritchie wrote:

Try rsync once, then judge for yourself. I'm totally convinced.

On Wed, Jul 18, 2012 at 3:59 PM, Robin Kraft <
reply@reply.github.com

wrote:

Here 'tis

You have to click to expand the comment.

It's still extremely convenient to have the repo on the cluster, for
quickly switching between branches, etc. I'm not convinced rsync is the way
to go.

---

Reply to this email directly or view it on GitHub:
#2 (comment)

Sam Ritchie, Twitter Inc
703.662.1337
@sritchie09

(Too brief? Here's why! http://emailcharter.org)

---

Reply to this email directly or view it on GitHub:
#2 (comment)

[robinkraft]

Are you using s3fs in conjunction with rsync?

http://superuser.com/a/433329

On Jul 18, 2012, at 4:02 PM, Sam Ritchie wrote:

Try rsync once, then judge for yourself. I'm totally convinced.

On Wed, Jul 18, 2012 at 3:59 PM, Robin Kraft <
reply@reply.github.com

wrote:

Here 'tis

You have to click to expand the comment.

It's still extremely convenient to have the repo on the cluster, for
quickly switching between branches, etc. I'm not convinced rsync is the way
to go.

---

Reply to this email directly or view it on GitHub:
#2 (comment)

Sam Ritchie, Twitter Inc
703.662.1337
@sritchie09

(Too brief? Here's why! http://emailcharter.org)

---

Reply to this email directly or view it on GitHub:
#2 (comment)

[robinkraft]

Oh, right, missed that detail.

On Jul 18, 2012, at 4:21 PM, Sam Ritchie wrote:

What? no, rsync the jar up to the jobtracker machine from your machine
after the deploy finishes and the cluster's ready. @eightysteele knows his
shit, he can help.

On Wed, Jul 18, 2012 at 4:15 PM, Robin Kraft <
reply@reply.github.com

wrote:

Are you using s3fs in conjunction with rsync?

http://superuser.com/a/433329

On Jul 18, 2012, at 4:02 PM, Sam Ritchie wrote:

Try rsync once, then judge for yourself. I'm totally convinced.

On Wed, Jul 18, 2012 at 3:59 PM, Robin Kraft <
reply@reply.github.com

wrote:

[Here 'tis](

wri/forma-clj#130 (comment))

You have to click to expand the comment.

It's still extremely convenient to have the repo on the cluster, for
quickly switching between branches, etc. I'm not convinced rsync is the
way
to go.

---

Reply to this email directly or view it on GitHub:

#2 (comment)

Sam Ritchie, Twitter Inc
703.662.1337
@sritchie09

(Too brief? Here's why! http://emailcharter.org)

---

Reply to this email directly or view it on GitHub:

#2 (comment)

---

Reply to this email directly or view it on GitHub:
#2 (comment)

Sam Ritchie, Twitter Inc
703.662.1337
@sritchie09

(Too brief? Here's why! http://emailcharter.org)

---

Reply to this email directly or view it on GitHub:
#2 (comment)

[robinkraft]

For reference, Sam's suggested solution, from here

1. Put a jar on S3 and download that as part of the deploy.
2. Build an uberjar locally and rsync it up to that location on the jobtracker.

The idea is that since rsync will only sync the changes, even if we're changing branches, etc., not that much is actually changing within the uberjar so it won't take long to sync.