Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Syano
Jul 13, 2005


Ok I need some help from some iscsi gurus and I figured this was the place to ask. I have an EMC CX3-10 array and a file server running win2k3 attached to it running powerpath 5.2 and iscsi initiator. This server is losing its shares every time it gets rebooted. No problem right? The lanman service is coming up before the iScsi targets are reconnected. Well, that is the problem. I have gone through the process of making the lanman service dependent upon the iscsi and powerpath services but I am still having the problem of the shares being down upon reboot and having to reshare the folders.

I have gone through Dell support and still have an active ticket with those guys but I thought maybe there was something I was missing that one of you guys could throw my way.

Adbot
ADBOT LOVES YOU

da sponge
May 24, 2004

..and you've eaten your pen. simply stunning.

Syano posted:

Ok I need some help from some iscsi gurus and I figured this was the place to ask. I have an EMC CX3-10 array and a file server running win2k3 attached to it running powerpath 5.2 and iscsi initiator. This server is losing its shares every time it gets rebooted. No problem right? The lanman service is coming up before the iScsi targets are reconnected. Well, that is the problem. I have gone through the process of making the lanman service dependent upon the iscsi and powerpath services but I am still having the problem of the shares being down upon reboot and having to reshare the folders.

I have gone through Dell support and still have an active ticket with those guys but I thought maybe there was something I was missing that one of you guys could throw my way.

Try to have the lanman service dependent on the spooler service. IIRC, it's one of the last services to come up.

Syano
Jul 13, 2005


I think I solved my own issue so I figured I would come post in case anyone would benefit. It appears that in the PowerPath 5.2 software the iSCSI targets show as "Powerpath Device" in the registry instead of "iSCSI Device" therefore they cannot be bound persistent with the iSCSI initiator. There is a registry fix available that involves basically adding a DWORD value. Of course I can't test it right now because I can't just reboot a file server willy nilly when I want to.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!


Hair Elf

Not sure if this is the best place to ask but should be close enough.

We run a test model environment where a bunch of projects come in and set up model networks etc and test them out before they go and build a live version.

All of this has been internal-company type stuff, but we're about to now go into a cost-recovery charging model where we'll charge the individual projects for the various model services.

One of the things my boss is interested in doing is providing a simple backup service that we can charge the projects for on a per TB basis or whatever, and is wondering what can be done in the way of a proof of concept.

He's currently thinking something in the realm of NZ$2000 (about US$1k) to start with, and eventually having us go to a larger setup once projects come in that have the requirement for a large amount of backup space.

What's a good way to set up a reasonably cheap NAS-like solution, preferably something that has a simple to use interface for management, and of course works across various OS's. Performance isn't a major factor as it'll only be for backups and recovery, not live storage. I was thinking a large server with a bunch of 1TB drives in it, but not sure what would be a good software for this. And ultimately we're only a test environment, not production, so there's nothing life and death about the solution.

skullone
Jan 28, 2009


Thought I'd provide an update on my Sun X4540 / Thor adventures.
I haven't had a lot of time to devote to it, so it's been sitting around as a $100k test environment... would've been my dream to play around with it had I not been so busy.

We had to use OpenSolaris 2008.11 for the in-kernel CIFS service, but it introduced us to a bit of a "void" with Sun support. Granted, we PAID for Solaris -and- OpenSolaris support contracts, when you call/e-mail support regarding OpenSolaris, the Sun support people are virtually useless. They know nothing about OpenSolaris, and still think it's not even released yet (my christ...).
So I've been doing all my own support related to ZFS snapshots, replication, iSCSI target software, network stack kernel panics, and on and on.

I'm really not happy with Sun themselves, but theres nowhere else you can get 100TB for $100k, even with some of the issues we're running into, its a good deal.

Most of my issues are resolved, or worked around, and we're seeing 100+MB/sec on CIFS reads and writes, and iSCSI performance is near wire-speed as well.
Though we did have to disable the ZIL, which puts us at a small risk for data loss, before it enters production I intend to install a separate file system log device on an SSD or something for performance.

I'm at the point now, that bonded GigE ports aren't enough, we can saturate 400MB/sec quite easily, and the Sun box isn't even sweating. I'm hoping with 10GigE, I can make the server get a good workout!

Overall, if you have Solaris/OpenSolaris experience, and some time to work out kinks in your environment, I'm pleased with our solution.
However, if you are faint of heart, or lack Solaris skills, stay away from Sun for the time being.

With risk, comes reward though. Or loss of your job.

complex
Sep 16, 2003



Hmm. We could not get out x4500 anywhere near wirespeed using iSCSI. Wondering now if the x4540 is that much better...

Any see my question on the last page on deduplication?

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.


complex posted:

Hmm. We could not get out x4500 anywhere near wirespeed using iSCSI. Wondering now if the x4540 is that much better...

Any see my question on the last page on deduplication?
What iSCSI target stack were you using, out of curiosity?

complex
Sep 16, 2003



Misogynist posted:

What iSCSI target stack were you using, out of curiosity?

Windows Server, not sure what version, 2003? I'm not a Windows guy.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.


complex posted:

Windows Server, not sure what version, 2003? I'm not a Windows guy.
I mean the target server running on the x4500 (unless you mean you were actually using Windows on the x4500 and I'm misunderstanding). The new COMSTAR stack in OpenSolaris has seen some pretty substantial improvements over the last few releases.

complex
Sep 16, 2003



Doh. Sorry. Target, not initiator.

Solaris 10 U4

1000101
May 14, 2003

BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY FRUITCAKE!

complex posted:

What kind of performance hit will deduplication incur?

Say I have 26 servers, A through Z. They all have 72G drives now, but they use ~10GB on each of them, and assume that ~3GB of that is exactly the same base OS image.

Dedeuplication will obviously save us a lot of space. I've seen the NetApp demo videos and it sounds awesome. But people are now telling me that performance will suffer. Still others say that all your duped blocks will probably be sitting in cache or on SSD anyway, so performance actaully increases.

It can perform very well. I have a customer with de-duped webservers (several hundred of them per filer) and he's seeing a huge savings on disk space. You can reduce the negative


quote:

I can see both sides of it: if I am just reading the same block all the time (say, a shared object in Linux or a DLL in Windows), then if that block is deduped then I'll be winning. But lets say I modify that block, then the storage array will sort of have to pull that block out and now start keeping a second copy of it, and managing that slows the array down.

Thoughts?

Its not so much the managing of that second block as it is too many servers actually trying to access that block and if you don't have it cached.

1000101
May 14, 2003

BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY FRUITCAKE!

Bumping this in the hopes that some EMC guys can tell me about RecoveryPoint and how it can give me application consistent replication/handles quiescing of snapshots.

Are you still watching?

Boner Buffet
Feb 16, 2006


I hate sales people. I want to get pricing on some lefthand gear, but I don't want to listen to any of their spiel, or get follow up calls only to have the person get pissy when I remind them we're a public school and everything comes down to dollars, not necessarily doing things the proper way. I don't want to talk to a reseller either. All I want to know is how much it costs.

Are there any resources available which might have this information available without having to talk to someone or does it pretty much come down to having to put up with sales people?

Maneki Neko
Oct 27, 2000



InferiorWang posted:

I hate sales people. I want to get pricing on some lefthand gear, but I don't want to listen to any of their spiel, or get follow up calls only to have the person get pissy when I remind them we're a public school and everything comes down to dollars, not necessarily doing things the proper way. I don't want to talk to a reseller either. All I want to know is how much it costs.

Are there any resources available which might have this information available without having to talk to someone or does it pretty much come down to having to put up with sales people?

I'll send you the numbers I've seen, but you don't seem to accept emails/pms.

Catch 22
Dec 1, 2003
Damn it, Damn it, Damn it!

1000101 posted:

Bumping this in the hopes that some EMC guys can tell me about RecoveryPoint and how it can give me application consistent replication/handles quiescing of snapshots.

Are you still watching?
You would get an app consistent snap first (using SnapView to manage and set this up) then RecoverPoint replicates at the blocklevel (clones) the LUN. Flatfiles would not need the snap first.

Edit: I also just shot off a email to my EMC guys to make sure there is not another way with the new SE version.

Catch 22 fucked around with this message at 14:48 on Mar 18, 2009

Boner Buffet
Feb 16, 2006


Maneki Neko posted:

I'll send you the numbers I've seen, but you don't seem to accept emails/pms.

I didn't realize that I had email turned off. I turned it on. No PM because I'm a cheap bastard who never purchased it.

Thanks!

Catch 22
Dec 1, 2003
Damn it, Damn it, Damn it!

InferiorWang posted:

I didn't realize that I had email turned off. I turned it on. No PM because I'm a cheap bastard who never purchased it.

Thanks!
I can help you cut some costs from EMC if you are looking at others besides Lefthand, if I understand what your needs are.

ozmunkeh
Feb 28, 2008

hey guys what is happening in this thread


Intrepid00 posted:

Update since we put Lefthand boxes in production...

They are awesome

Users are starting to notice the increased preformance as crap is moved off the DAS.

Just out of interest which boxes did you go for, and how many apps/users are they supporting?

H110Hawk
Dec 28, 2006


InferiorWang posted:

I hate sales people. I want to get pricing on some lefthand gear, but I don't want to listen to any of their spiel, or get follow up calls only to have the person get pissy when I remind them we're a public school and everything comes down to dollars, not necessarily doing things the proper way. I don't want to talk to a reseller either. All I want to know is how much it costs.

Are there any resources available which might have this information available without having to talk to someone or does it pretty much come down to having to put up with sales people?

Get a full retail quote from them for "the right way", have them itemize costs as much as they can claiming government red tape. Then, generate quote for what you actually want, cut price 50%, and make up a PO. Call them up and send them your proposal and tell them it's this or EMC. See what happens. You generally won't be able to cut the service contract by nearly 50%, and I assume this is required by your school policy.

If they still balk, remind them that you have to keep a service contract on the device for 3? 5? years due to that same government red tape. Don't be afraid to lie outright to them, their sales guys will do the same to you.

H110Hawk fucked around with this message at 16:14 on Mar 18, 2009

Catch 22
Dec 1, 2003
Damn it, Damn it, Damn it!

H110Hawk posted:

Get a full retail quote from them for "the right way", have them itemize costs as much as they can claiming government red tape. Then, generate quote for what you actually want, cut price 50%, and make up a PO. Call them up and send them your proposal and tell them it's this or EMC. See what happens. You generally won't be able to cut the service contract by nearly 50%, and I assume this is required by your school policy.

If they still balk, remind them that you have to keep a service contract on the device for 3? 5? years due to that same government red tape. Don't be afraid to lie outright to them, their sales guys will do the same to you.

You can also save by doing the install yourself (if they are charging for it) and if you get the Manufacture behind you they can discount the Vendors quote passing savings to you. In my case I got a discount and a extra warranty year for nothing. Call Lefthand and talk with them. It can pay off.

H110Hawk
Dec 28, 2006


Catch 22 posted:

You can also save by doing the install yourself (if they are charging for it) and if you get the Manufacture behind you they can discount the Vendors quote passing savings to you. In my case I got a discount and a extra warranty year for nothing. Call Lefthand and talk with them. It can pay off.

Self-install of most stuff is a snap, assuming you aren't afraid of lifting disk trays. Just make sure a self-install doesn't conflict with your service contract and warranty. If it does, they should do it for free since you are paying for it with the service contract.

Boner Buffet
Feb 16, 2006


Thanks fellas. What I'd really like to do is have an iSCSI SAN, 2-4 TB. I'd like to host a modest amount of vmware guests on it hosting primarily file shares, home directories, and groupwise email for the staff numbering 250-300. Then I'd like to do it all over again with a completely redundant SAN at one of our other schools in some sort of fail over configuration. Being able to do snapshots and have some sort of monitoring dashboard is something I'd like as well.

My problem is I can't even get anyone to listen to me about doing this and getting away from bare metal machines for our critical data without having some semblance of a dollar figure attached to it. I have to approach this rear end backwards from how most normal people would approach it.

1000101
May 14, 2003

BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY FRUITCAKE!

Catch 22 posted:

You would get an app consistent snap first (using SnapView to manage and set this up) then RecoverPoint replicates at the blocklevel (clones) the LUN. Flatfiles would not need the snap first.

Edit: I also just shot off a email to my EMC guys to make sure there is not another way with the new SE version.

You happen to have any links to EMC papers that talk about SnapView integration and how it works to quiesce data stored on a VMFS volume?

Google has me coming up clean and all the VMware integration papers have pretty much this to say:

"SnapView makes clones!"

But it doesn't tell me how it handles application quiescing.

Something else has pointed me to yet another product (Replication Manager) but I want to make sure there isn't something else I need.

Vanilla
Feb 24, 2002

Hay guys what's going on in th

So what Recoverpoint does is, as far as i'm aware, is use Replication Manager as the tool that makes the snaps and clones. Replication Manager has been around for years and is a standalone product but is integrated / bundles when Recoverpoint is sold.

Replication manager manages the snaps / clones on the array as it has the integration brains with regards to Exchange, SQL, VMWare, etc. Array based snap tools are SNapview (Clariion) or Timefinder Snap (DMX) When it comes to application such as exchange it uses VSS, with Orcale it puts the DB in hot backup mode, etc.

Here is a blog that discusses the two options open to you - array based snapshots or the VM flavour

http://virtualgeek.typepad.com/virtual_geek/2009/02/howto---vmware-integrated-and-application-integrated-array-replicas.html

Let me know if anything isn't clear, i've dug the above link out real quick as i've a million things to do

rage-saq
Mar 21, 2001

Thats so ninja...

InferiorWang posted:

Thanks fellas. What I'd really like to do is have an iSCSI SAN, 2-4 TB. I'd like to host a modest amount of vmware guests on it hosting primarily file shares, home directories, and groupwise email for the staff numbering 250-300. Then I'd like to do it all over again with a completely redundant SAN at one of our other schools in some sort of fail over configuration. Being able to do snapshots and have some sort of monitoring dashboard is something I'd like as well.

My problem is I can't even get anyone to listen to me about doing this and getting away from bare metal machines for our critical data without having some semblance of a dollar figure attached to it. I have to approach this rear end backwards from how most normal people would approach it.

Pick a big number and get them to approve that, then show off how good you are by coming in under budget!

Maneki Neko
Oct 27, 2000



InferiorWang posted:

Thanks fellas. What I'd really like to do is have an iSCSI SAN, 2-4 TB. I'd like to host a modest amount of vmware guests on it hosting primarily file shares, home directories, and groupwise email for the staff numbering 250-300. Then I'd like to do it all over again with a completely redundant SAN at one of our other schools in some sort of fail over configuration. Being able to do snapshots and have some sort of monitoring dashboard is something I'd like as well.

My problem is I can't even get anyone to listen to me about doing this and getting away from bare metal machines for our critical data without having some semblance of a dollar figure attached to it. I have to approach this rear end backwards from how most normal people would approach it.

You might be able to get pricing guesstimates on the Lefthand stuff (since email doesn't seem to be working on the forums at the moment) by googling part numbers from here: http://www.gosignal.com/datasheets/lefthandsan.pdf

2-4 TB isn't much storage wise, my ballpark guess for the SATA starter edition with installation would be around 30k, but you should hopefully be able to negotiate that down.

Do you have a CDW rep? If not, call and get one ask for pricing on those part numbers. Those guys are frothing at the mouth for sales right now.

1000101
May 14, 2003

BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY FRUITCAKE!

Vanilla posted:

So what Recoverpoint does is, as far as i'm aware, is use Replication Manager as the tool that makes the snaps and clones. Replication Manager has been around for years and is a standalone product but is integrated / bundles when Recoverpoint is sold.

Replication manager manages the snaps / clones on the array as it has the integration brains with regards to Exchange, SQL, VMWare, etc. Array based snap tools are SNapview (Clariion) or Timefinder Snap (DMX) When it comes to application such as exchange it uses VSS, with Orcale it puts the DB in hot backup mode, etc.

Here is a blog that discusses the two options open to you - array based snapshots or the VM flavour

http://virtualgeek.typepad.com/virtual_geek/2009/02/howto---vmware-integrated-and-application-integrated-array-replicas.html

Let me know if anything isn't clear, i've dug the above link out real quick as i've a million things to do

So what its looking like is that if I want to take consistent snapshots of VMFS volumes, I need to buy Replication Manager correct?

I presume it works by telling vCenter to take vmware snapshots (kicking off VSS in the guests) and when that completes it fires off an array based snapshot? (This is how SnapManager on NetApp works).

The EMC engineer assigned to this account is pretty worthless when it comes to volunteering information and english isn't his first language which only makes things worse.

edit: Many thanks for the blog entry, I see there is a cellera VSA there which will make my life about 1000 times easier.


Vanilla
Feb 24, 2002

Hay guys what's going on in th

1000101 posted:

So what its looking like is that if I want to take consistent snapshots of VMFS volumes, I need to buy Replication Manager correct?

I presume it works by telling vCenter to take vmware snapshots (kicking off VSS in the guests) and when that completes it fires off an array based snapshot? (This is how SnapManager on NetApp works).

The EMC engineer assigned to this account is pretty worthless when it comes to volunteering information and english isn't his first language which only makes things worse.

edit: Many thanks for the blog entry, I see there is a cellera VSA there which will make my life about 1000 times easier.

Correct, get the local engineer / TC to supply you with some documentation. He needs to qualify that RM will do what you want and work in yuor environment before you buy.

Have a play with the 90 day demo.

It's a great blog for anything EMC and VMware. I'm really getting into the VDI stuff personally.

skipdogg
Nov 29, 2004
Resident SRT-4 Expert


Vanilla posted:

I'm really getting into the VDI stuff personally.

Though my environment would never justify the costs, VDI is loving awesome. I can't wait to see what they do with laptops. eWeek had a neat article about hypervisors running on laptops.. With an extremely blurry line between personal and business use of laptops, there's only so much NAC, whitelisting, etc you can do without frustrating IT and your users...

1000101
May 14, 2003

BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY FRUITCAKE!

Vanilla posted:

It's a great blog for anything EMC and VMware. I'm really getting into the VDI stuff personally.

I'm currently at one of the largest retailers in the world doing a VMware design that specifically will be leveraging VDI and EMC storage. I want this to be as big a win as humanly possible for VMware and EMC since it will give us no end of clout with both organizations.

It's also going to look great for me professionally.

As such, I'm trying to dig up every word of information I can cram into my skull regarding EMC DR/BC, SRM integration, VMware integration, etc.

My biggest complaint is that the guy who wrote the VMware whitepaper (h1416-emc-clariion-intgtn-vmware-wp) is either brain dead or has no idea how VMware works or how people are implementing VMware in their environments. I can't give this paper to a customer in the state it's currently in.

Now that I know roughly what I'm looking for, I'm finding a decent amount of information though. I'll probably have Replication Manager running in my lab next week.

H110Hawk
Dec 28, 2006


InferiorWang posted:

My problem is I can't even get anyone to listen to me about doing this and getting away from bare metal machines for our critical data without having some semblance of a dollar figure attached to it. I have to approach this rear end backwards from how most normal people would approach it.

Remember above where I said lie to them? Do that. I imagine at this point they're calling you and wanting to setup meetings? Tell them to bring their spreadsheets because you have to talk cost as well as performance. Let them go through their whole dog and pony show with the glossy brochures and powerpoint, then ask them what "that" (pointing at the screen) costs. Play hardball right back.

Boner Buffet
Feb 16, 2006


rage-saq posted:

Pick a big number and get them to approve that, then show off how good you are by coming in under budget!

I might be better off saying it's cheap and ending up over budget. Otherwise I'm likely to get this response:

I'll look into the CDWG route. I do have a rep and he's generally pretty good.

rage-saq
Mar 21, 2001

Thats so ninja...

InferiorWang posted:

I might be better off saying it's cheap and ending up over budget. Otherwise I'm likely to get this response:

I'll look into the CDWG route. I do have a rep and he's generally pretty good.

I'd advise against getting any kind of advice on important system design from what is essentially the best buy of product fulfillment centers. Call in a consultant and engage with them in a discovery and project assessment for virtualizing your infrastructure.

brent78
Jun 23, 2004

I killed your cat, you druggie bitch.

rage-saq posted:

I'd advise against getting any kind of advice on important system design from what is essentially the best buy of product fulfillment centers.
This. Never buy hardware from CDW unless it's parts or software. Find a reseller that's certified in the solution you're interested in and has good pre-sales engineering support that can design solution. CDW is essentially NewEgg with account reps. CDW also won't be able to get you the lowest pricing. With that said, CDW does have a nice interface for managing software licensing like Adobe, Microsoft, etc..

Mierdaan
Sep 14, 2004



Pillbug

rage-saq posted:

I'd advise against getting any kind of advice on important system design from what is essentially the best buy of product fulfillment centers. Call in a consultant and engage with them in a discovery and project assessment for virtualizing your infrastructure.

Check out my post history in this thread if you want to see why you shouldn't do this.

Catch 22
Dec 1, 2003
Damn it, Damn it, Damn it!

Mierdaan posted:

Check out my post history in this thread if you want to see why you shouldn't do this.

Ha, yes. I remember watching that unfold.

Mierdaan posted:

Well, we've been hosed.
<stuff>

rage-saq posted:

Reason number 5234634 to not deal with those big fullfilment warehouses like CDW, PC-Mall etc for configs.

I used CDW as a price wedge to Dell, and had EMC come in first hand to help, even using EMC to force Dells price point down. I love pitting vendors against each other in this current economy. They get vicious. I got our last SAN for a steal.

With that said, do LOTs of homework. I spent 3 months prepping, getting performance metrics and even a few class seminars (not held by vendors, but by a partnership of peer firms, vendor held ones are all sales gimmicks and might drive you away from a better product)
Its worth your time, as to not run into issues down the road.

Catch 22 fucked around with this message at 15:32 on Mar 20, 2009

skullone
Jan 28, 2009


complex posted:

Windows Server, not sure what version, 2003? I'm not a Windows guy.

I use the OpenSolaris COMSTAR iSCSI target, with both Win2k3 and Win2k8 iSCSI initiators.
Note: I DID disable the ZIL. I saw terrible numbers with it enabled (almost as slow as entry level EMC/BluArc)

I'll get some DRAM ZIL device someday... I hope

brent78
Jun 23, 2004

I killed your cat, you druggie bitch.

I have 4 Coraid SR-1521's with 15 x 500GB drives each (7.5 TB RAW each), willing to unload real cheap to the first people who PM me. I was using them for a media project and now no longer need them.

oblomov
Jun 20, 2002

Meh... #overrated

Catch 22 posted:

You would get an app consistent snap first (using SnapView to manage and set this up) then RecoverPoint replicates at the blocklevel (clones) the LUN. Flatfiles would not need the snap first.

Edit: I also just shot off a email to my EMC guys to make sure there is not another way with the new SE version.

I just saw a presentation by EMC last week replicating Oracle RAC and it was pretty drat impressive. Now, it looks like managing the various caches may be a bit of a hassle, and it's the usual EMC CX4 stuff (I still like NetApp better), but for replicating copious amounts of data, this is some very good stuff. Cool part is they track the changes and only replicate changed block at certain interval (with Async), say like every 20-30 min, so unlike SnapMirror, the amount of data after compressions and folded writes (i think that's what the EMC engineer called the process) appears to be going 3-1 for Oracle. Not too shabby. That said, it's not cheap and requires whole bunch of fiber connections.

Adbot
ADBOT LOVES YOU

H110Hawk
Dec 28, 2006


brent78 posted:

I have 4 Coraid SR-1521's with 15 x 500GB drives each (7.5 TB RAW each), willing to unload real cheap to the first people who PM me. I was using them for a media project and now no longer need them.

I might take them. Shoot me a PM with your initial cost quote. How used are they?

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply