Living in an Ivory Basement Stochastic thoughts on science, testing, and programming.

By C. Titus Brown · before

:::bash
mkdir subset
for i in data/*.fastq
do
    head -400 $i > subset/$(basename $i).subset.fastq
done

>data/F3D0_S188_L001_R1_001.fastq
>data/F3D0_S188_L001_R2_001.fastq
>...

>subset/F3D0_S188_L001_R1_001.fastq.subset.fastq
>subset/F3D0_S188_L001_R2_001.fastq.subset.fastq

:::bash
mkdir subset2
for i in data/*.fastq
do
    head -400 $i > subset2/$(basename $i .fastq).subset.fastq
done

:::python
# pull in all files with .fastq on the end in the #data
FILES = glob_wildcards('data/{name}.fastq')

# extract the {name} values into a list
NAMES = FILES.name

rule all:
    input:
        # use the extracted name values to build new filenames
        expand("subset3/{name}.subset.fastq", name=NAMES)

rule subset:
    input:
        "data/{n}.fastq"
    output:
        "subset3/{n}.subset.fastq"
    shell: """
        head -400 {input} > {output}
    """

:::python
# pull in all files with .fastq on the end in the 'data' directory.             
PREFIX = config.get('prefix', 'data')
print(f"looking for FASTQ files under '{PREFIX}'/")

OUTPUT = config.get('output', 'subset5')
print(f"subset results will go under '{OUTPUT}'/")

FILES = glob_wildcards(f'{PREFIX}/{{name}}.fastq')

# extract the {name} values into a list                                         
NAMES = FILES.name

# request the output files                                                      
rule all:
    input:
        # use the extracted 'name' values to build new filenames                
        expand("{output}/{name}.subset.fastq", output=OUTPUT, name=NAMES)

# actually do the subsetting                                                    
rule subset_wc:
    input:
        f"{PREFIX}/{{n}}.fastq"
    output:
        "{output}/{n}.subset.fastq"
    shell: """                                                                  
        head -400 {input} > {output}                                            
    """

fasta_file soil-data.fa

qc_filter min_length=50 remove_Ns=true

graph_filter min_length=400

velvet_assemble k=33 min_length=1000 scaffolding=True

CloudStore (KFS, previously Kosmosfs) is Kosmix's C++ implementation of
Google File System. ... CloudStore supports incremental scalability,
replication, checksumming for data integrity, client side fail-over and access
from C++, Java and Python.

hey, i used that whilst studing TDD for a CMPT 376 (tech writing)
paper.

Great link, thanks!

foo

i'm going to be recoding <x> from scratch starting next week, in python.
what codebase would you recommend as good to model after?

It turns out that while the Python stdlib supports …
"Penis size" style of programming -- the (mistaken) belief that the
more advanced programming language features you use, the more
impressive your code will look.

"""
A-grade engineers are unfortunately similar to Welsh
longbowmen: devastatingly potent compared to their peers, but you
have to start their training at age 10 or so.  Simply upping the
salaries of A-grade engineers won't magically create more of them.
We know this, as we …
idyll …
As we pass the one thousand databases mark (1kDB) I wonder, what
proportion of the data in these databases will never be used?

print 'hello …
class …
Trying to understand what massive pools of threads that spawn other
massive spools of threads, that spawn other massive pools of threads …
page_obj = parse(html_code)

form_obj = page_obj.forms[0]

form_obj.my_name = 'bob'
form_obj.my_select.choose('option2')

new_html_code = page_obj.emit_html()

Practical Agile Web Testing
---------------------------

Have Web site?  Need testing?  Bring your tired (code …
sys.settrace
sys.stdout`, then don't ever trash ``sys.stdout
Great article, at least it is a comfort that some people admit it is
hard and that "software is hard" is becoming more …
Publications relying on derivative works of this software must …
I …
Hmm, I wouldn't call "better than Perl" validation. "Easier to read
the Perl, better threading support than COBOL, more productive than
TriMedia VLIW assembler, faster than MUMPS …
Um... faculty members... programming? You lost …
"Don't write a test plan. Instead, test."

             -- Bret Pettichord

random.shuffle
import random
l = range(0, 20) # array to permute

for j in range(1, len(l)):
   k = random.randint(0, j)
   l[j], l …
subprocess
commands
subprocess
output = get_output(cmd, input=None, cwd=None, env=None …
unittest
unittest
# Demo 1: Testing CherryPy
# Demo 2: Testing CherryPy without exec'ing a process
# Demo 3: Basic code coverage analysis with figleaf
# Demo …
class TestFoobar:
    def test_is_a_singleton(self):
        pass
    def test_can_be …
figleaf.start_section(name …
fork()
subprocess
yield
def test()
   for i in range(0, 10):
      yield check …
Never heard of DART, but I used something called STAF/STAX in the
past, and I had to struggle with tests defined in XML syntax …
I think we should play some GalCon at PyCon this year! (Your earlier
post about GalCon got me completely addicted.)    Take care,    -Brian

...twill is more or less Py web-developer 101 at this point, so a talk
about it should be enlightening.

There are several de facto models of open source at this level (the
small niche projects level, that is).

The first is …
extend_with twill_xmpp

jabber_login titus2@xmpp.us XXXXX
send_jabber_message titus@xmpp.us message

#! /usr/bin/env python
import sys, email
from pyxmpp.jid import JID
from pyxmpp.jabber.simple import send_message

inp = sys.stdin.read()
message = email.message_from_string(inp)

jid = 'someid@xmpp.us'
password, recpt = 'XXXXX', 'otherid@xmpp.us' …
ocomments
ocomments
comments

Other articles

General background

People are hungry for hard and interesting problems

Introduction